POC of integration receiver and processor #96
Conversation
|
This looks really promising! I think having the templates as a receiver solves two main issues compared to the "Converter" approach:
|
There was a problem hiding this comment.
Thanks for adding this! I wanted to check if a receiverscreator would also work, template config:
extensions:
host_observer:
receivers:
receiver_creator/2:
# Name of the extensions to watch for endpoints to start and stop.
watch_observers: [host_observer]
receivers:
httpcheck/on_host:
# If this rule matches an instance of this receiver will be started.
rule: type == "port" && port == 8080
resource_attributes:
service.name: redis_on_host
processors:
attributes/example:
actions:
- key: account_id
value: 2245
action: insert
pipelines:
extensions: host_observer
metrics:
receiver: ""
processors:
- attributes/exampleFor the previous configuration to work we would need to add support for extensions in the configuration. I think that supporting extensions could be added in the future, we can focus on templates with just plain receivers + processors for the moment.
pkg/templates/templates.go
Outdated
| } | ||
|
|
||
| type Config struct { | ||
| Receivers map[component.ID]map[string]any `mapstructure:"receivers"` |
There was a problem hiding this comment.
Some receivers like the receivecreator relies on extensions, see: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/receivercreator#examples
Could we include the extension component in the config?
There was a problem hiding this comment.
Added open question about this to the description, it might make sense for authentication in some cases. For others it might be better to let the option to users to define any extension they want.
There was a problem hiding this comment.
Hmm I'm not sure I get it, but the receivercreator would not be loaded as an integration receiver right? Hence its dependency to the observer extension does not really matter here?
However that's a valid point because we will aim to create technology specific integrations to cover logs use-cases then we would most probably need to pair filelogreceiver with the storage extension. See https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/examples/fault-tolerant-logs-collection/README.md and https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/examples/fault-tolerant-logs-collection/otel-col-config.yaml#L4.
There was a problem hiding this comment.
Yes, what I mean is that if an integration requires an extension, the integration could expose a variable, and the extension would need to be defined by the user in the config.
This may look as requiring additional configuration, but also allows to have simpler and more composable integrations.
Look for example to the POC for a nginx integration @mrodm prepared.
Templates include placeholders for the storage to use:
https://github.com/elastic/integrations/pull/11253/files#diff-26a3b4eab2c9b845b76591d786a09cd833ac1527288dd2e0c4e2c457fc853521R93
filelog/access:
include_file_path: true
include: ${var:access_paths}
start_at: beginning
storage: ${var:storage_resource}
The final users needs to define an extension, and set the parameter for the integration: https://github.com/elastic/integrations/pull/11253/files#diff-7187c0dbde58e198aa05659ae047693f9fa1a12fb16cebece7e7ea06602901b1
extensions:
...
file_storage:
directory: ./data
receivers:
integration/nginx_otel:
name: "nginx_otel"
parameters:
...
storage_resource: file_storage
I think this is a nice approach, because this way the user can reuse a single extension definition for many receivers, being them integrations or not.
There was a problem hiding this comment.
Thank's for elaborating, that makes sense!
pkg/templates/templates.go
Outdated
| Pipelines map[component.ID]PipelineConfig `mapstructure:"pipelines"` | ||
| } | ||
|
|
||
| func (c *Config) Validate() error { |
There was a problem hiding this comment.
Is this function run during the collector's lifetime? I reckon the component's Validate interface implementation is run by the collector at startup. As this is not a component, I think this Validate function won't be run, should we move this check to a custom UnMarshal function instead?
pkg/templates/templates.go
Outdated
| Processors []component.ID `mapstructure:"processors"` | ||
| } | ||
|
|
||
| func (p *PipelineConfig) Validate() error { |
There was a problem hiding this comment.
I think this function won't be run
There was a problem hiding this comment.
I will take a look to this.
jsoriano
changed the title
Summary
This POC revisits the idea exposed in the open-telemetry/opentelemetry-collector-contrib#26312 to use a receiver to render templates as a mechanism to distribute reusable configurations as part of integrations. Templates are in this POC more similar to OTel collector configuration files, with placeholders for variables.
This POC includes the following components:
internal/integrationswith helper code to work with integrations in other components.extension/fileintegrationextensionwith an extension that can be used by other components to obtain integrations from local disk.extension/configintegrationextensionwith an extension that can be used to embed integration templates in the main OTel collector configuration.processor/integrationprocessorprocessor that is composed by instantiating processors from a integration.receiver/integrationreceiverreceiver that is composed by instantiating receivers and processors from a integration.Open questions:
Purpose and use-cases of the new components
The main purpose of integrations is to provide an abstraction to share configurations intended to observe specific applications and services.
Templates are interpreted by components created with this purpose. Two main components are implemented for that, a receiver and a processor. They can use extensions to discover integrations.
These components are intended to be used with minimal configuration, so users can leverage integrations without extensive knowledge of the OTEL collector. The minimal configuration users need to provide is one or more sources of integrations, configured as extensions, the name of the integrations and the parameters it needs.
The use of receiver as the main component to use integrations will allow to leverage any other feature built for receivers, such as autodiscover features.
The main use case is the distribution of configurations for specific services, these configurations include receivers and processors, and pipelines to indicate how the receivers and processors have to be combined. A single integration can include multiple pipelines, users can chose which ones to use.
Find below an example integration, mostly copied from the one in open-telemetry/opentelemetry-collector#8372.
Example configuration for the component
Using the integration receiver:
Using the integration processor:
Telemetry data types supported
These are generic components that support any data type provided by other components.
Is this a vendor-specific component?
It is not vendor-specific.
Code owners
Elastic?
Sponsor
Elastic?
Additional context
This proposal is a follow-up of open-telemetry/opentelemetry-collector-contrib#26312 and open-telemetry/opentelemetry-collector#8372.